Skip to content

Add tile-backed proof generation - #54

Merged
Amaury Chamayou (achamayou) merged 10 commits into
mainfrom
achamayou/tiles-proofs
Aug 6, 2026
Merged

Add tile-backed proof generation#54
Amaury Chamayou (achamayou) merged 10 commits into
mainfrom
achamayou/tiles-proofs

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

Part 3 of the tiled-storage work. This branch is rebased onto current main (6d70d02) after #52 and #53 merged.

Compatibility

Tile hashes and proof assembly use the tree's existing HASH_FUNCTION; leaf and node hashing behavior is unchanged. Inclusion proofs remain byte-for-byte compatible with existing merklecpp paths, and the default SHA-256 implementation remains OpenSSL-independent.

CI

CMake detects each runner's logical processor count and exports the standard build and CTest parallelism variables. Tests use CMake's default-on BUILD_TESTING option. tiles_level2 remains behind LONG_TESTS, which is enabled for every Release matrix job. The test retains a 900-second timeout on Linux and a 3600-second timeout on Windows. CodeQL also builds the long-test target.

Rebase status

  • base: main at 6d70d02;
  • head: achamayou/tiles-proofs at 741fb70;
  • divergence: 0 commits behind, 10 commits ahead.

Validation

  • focused tiles_proofs and tiles_writer tests;
  • Debug Clang 18 ASan/UBSan tiles_proofs test;
  • Release build with LONG_TESTS=ON;
  • full parallel test suite (17/17 passed on Linux, including tiles_level2);
  • SHA-384 and SHA-512 reducer equivalence probes;
  • CI-equivalent clang-tidy checks for merklecpp.h and merklecpp_tiles.h.

Add HashSourceT (abstract subtree-root resolver), TileHashSourceT
(resolves from full tiles, with an LRU tile cache), ProofEngineT
(inclusion/consistency proofs and their verifiers, built on mth_range),
MemoryHashSourceT (resolves from a resident in-memory tree), and
CombinedHashSourceT (memory first, falling back to tiles).

Add the sole essential core change: TreeT::subtree_root(), a read-only,
non-hashing accessor that lets proofs be served from the resident tree,
plus the <limits> include it needs. No other merklecpp.h changes.

Add tiles_proofs tests, cross-checking tile-derived inclusion and
consistency proofs against merkle::TreeT as the oracle across a range
of sizes, including tile-boundary crossings. Move the memory-only
subtree_root proof coverage and the ProofEngineProbe hostile-arithmetic
edge cases here from tree coverage, since they exercise the proof
engine and core accessor rather than TiledTree lifecycle. Add
tiles_level2 for end-to-end coverage of the level-2 tile path.

Introduce the LONG_TESTS CMake option, gate tiles_level2 behind it, and
enable it in CI (and CodeQL) so long-running tile coverage runs on pull
requests.

Document the compatibility statement, the optional core accessor, and
the HashSource/ProofEngineT API and algorithms in the design doc.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the current default tree hash function for public proof aliases, reject index conversions that cannot be represented safely, and align the design and level-2 coverage notes with the rebased implementation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c504572c-322f-4950-8682-edf4a7fd2c5b
@achamayou
Amaury Chamayou (achamayou) changed the base branch from achamayou/tiles-writers to main July 25, 2026 21:13
@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review August 3, 2026 13:13
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner August 3, 2026 13:13
Copilot AI review requested due to automatic review settings August 3, 2026 13:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

There are actionable API/performance issues in the new proof/tile-reading code paths that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR adds tile-backed proof generation infrastructure for merklecpp’s tiled-storage stack, including abstract subtree-root sources (memory, tiles, and combined), a proof engine that produces inclusion/consistency proofs compatible with existing TreeT paths, and test/CI wiring to validate large tile hierarchies.

Changes:

  • Add TreeT::subtree_root() to expose resident perfect-subtree roots without changing hashing semantics.
  • Introduce HashSourceT implementations and ProofEngineT for roots, inclusion proofs, RFC 6962 consistency proofs, and verification.
  • Add new proof and level-2 tile end-to-end tests plus LONG_TESTS gating and CI configuration to run long tests selectively.
File summaries
File Description
test/tiles_proofs.cpp Adds tile+memory proof equivalence tests vs existing TreeT::path() / past_path() and consistency verification.
test/tiles_level2.cpp Adds an end-to-end long test that materializes a full level-2 tile and validates roll-ups and TileHashSource resolution.
test/CMakeLists.txt Registers new tiles proof tests and gates tiles_level2 behind LONG_TESTS with a timeout.
merklecpp.h Adds TreeT::subtree_root() accessor for resident perfect subtrees with overflow/flush checks.
merklecpp_tiles.h Adds HashSourceT abstraction, tile/memory/combined implementations, and ProofEngineT (inclusion + consistency proofs).
doc/design/tlog-tiles.md Updates design doc to reflect the proof engine/hash-source architecture and lifecycle plan changes.
CMakeLists.txt Introduces LONG_TESTS CMake option.
.github/workflows/codeql-analysis.yml Enables LONG_TESTS during CodeQL build configuration.
.github/workflows/ci.yml Runs LONG_TESTS only on a single selected Linux job to avoid Windows timeouts.
Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 3
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread merklecpp_tiles.h Outdated
Comment thread test/tiles_proofs.cpp Outdated
Comment thread merklecpp_tiles.h

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

Proof-generation hot-path code introduces avoidable heap allocations, and the new tests rely on transitive standard header includes that should be made explicit.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

merklecpp_tiles.h:1025

  • roll_up() allocates a temporary std::vector slice for every subtree roll-up, which is a hot path during proof generation and defeats the goal of avoiding redundant copies. Since span is always a power-of-two <= TILE_WIDTH, this can be rolled up in-place using a fixed buffer and HASH_FUNCTION without heap allocation.
        }
        return perfect_root<HASH_SIZE, HASH_FUNCTION>(std::vector<Hash>(
          tile.begin() + (std::ptrdiff_t)off,
          tile.begin() + (std::ptrdiff_t)(off + span)));

test/tiles_proofs.cpp:15

  • This test throws std::runtime_error but does not include directly. Relying on transitive includes from merklecpp headers is not guaranteed and can break with different standard library implementations or include order.
#include <iostream>
#include <limits>
#include <merklecpp.h>
#include <merklecpp_tiles.h>
#include <string>

test/tiles_level2.cpp:18

  • This test throws std::runtime_error but does not include directly. Relying on transitive includes from other headers is not guaranteed and can break with different standard library implementations or include order.
#include <ctime>
#include <filesystem>
#include <iostream>
#include <merklecpp.h>
#include <merklecpp_tiles.h>
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new subtree-root accessor introduces a confirmed UB path (via is_full() shifting) and the new tests’ temp-dir naming can collide under the newly enabled parallel CI test execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

merklecpp_tiles.h:1025

  • TileHashSourceT::roll_up() allocates/copies a temporary std::vector<Hash> for every span>1 roll-up. This can become a noticeable hotspot during proof generation (many small roll-ups). You can compute the perfect root directly over the tile slice without allocating.
        }
        return perfect_root<HASH_SIZE, HASH_FUNCTION>(std::vector<Hash>(
          tile.begin() + (std::ptrdiff_t)off,
          tile.begin() + (std::ptrdiff_t)(off + span)));
      }

test/tiles_proofs.cpp:281

  • The temp directory name is derived from time() + rand() seeded with time(). With ctest --parallel enabled in CI, multiple test binaries can start in the same second and produce the same directory name, causing flaky collisions. Use a nonce that includes the process id and a monotonic/sequence component.
  const fs::path base = fs::temp_directory_path() /
    ("merklecpp_tiles_proofs_" + std::to_string((unsigned long long)seed) +
     "_" + std::to_string(std::rand()));

test/tiles_level2.cpp:50

  • The temp directory name is based only on time(nullptr), which can collide when tests run in parallel (CI now uses ctest --parallel). Use a name that incorporates process id and a monotonic/sequence nonce to avoid flaky collisions.
  const fs::path dir = fs::temp_directory_path() /
    ("merklecpp_tiles_level2_" +
     std::to_string((unsigned long long)std::time(nullptr)));

  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread merklecpp.h

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The CMake testing toggle change can silently break existing builds that still pass -DTESTS=ON/OFF, so compatibility mapping (or equivalent) should be added before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CMakeLists.txt:10

  • The build option for enabling/disabling tests switched from the project-specific TESTS cache variable to CTest's BUILD_TESTING. As written, -DTESTS=ON/OFF is now silently ignored (and BUILD_TESTING defaults to ON via include(CTest)), which can break existing scripts and make it hard to turn tests off in downstream builds. Consider adding a small compatibility shim that maps TESTS to BUILD_TESTING (with a deprecation warning) before including CTest.
include(GNUInstallDirs)
include(CTest)

  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The switch to BUILD_TESTING without top-level gating can unexpectedly enable tests and FetchContent downloads when merklecpp is consumed as a subproject, which can break downstream builds.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CMakeLists.txt:49

  • include(CTest) makes BUILD_TESTING default to ON, so this condition will add the test/ subtree (and FetchContent-download doctest) even when merklecpp is consumed as a dependency via add_subdirectory/FetchContent. That can unexpectedly pull network dependencies and slow or break downstream builds; consider only adding tests when merklecpp is the top-level project.
if(BUILD_TESTING)
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@achamayou

Amaury Chamayou (achamayou) commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

The switch to BUILD_TESTING without top-level gating can unexpectedly enable tests and FetchContent downloads when merklecpp is consumed as a subproject, which can break downstream builds.

There aren't downstream builds, we are doing this cleanup now.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Human review recommended

It changes core merkle tree internals and introduces new proof/verification logic plus CI/test wiring, which warrants final human review despite the added test coverage.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread .github/workflows/ci.yml
Comment thread merklecpp.h Outdated
Co-authored-by: eddyashton <6000239+eddyashton@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

The changes are cohesive, well-tested via new targeted and long-running tests, and the build/CI updates align with the new testing and proof-generation components without introducing evident correctness issues.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@achamayou
Amaury Chamayou (achamayou) merged commit f512faf into main Aug 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants